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1.  SUMMARY 

This  preliminary  report  focuses  on  research  and  development  (R&D)  and  demonstration  of  an 
unmanned  ground  system  (UGS)  technology  suitable  for  teleoperated  and  autonomous  robotic 
firefighting.  The  R&D  effort  was  conducted  by  the  Air  Force  Research  Laboratory’s  (AFRL), 
Materials  and  Manufacturing  Directorate,  Airbase  Technologies  Division  (RXQ),  Airbase 
Engineering  Development  Branch  (RXQE).  Common  field  testing  and  demonstrations  were 
conducted  at  Tyndall  AFB,  FL,  in  accordance  with  the  Department  of  Defense  (DoD) 
firefighting  concept  of  operation  (CONOPS).  In  addition,  a  demonstration  was  perfonned  at  Ft. 
Drum,  NY,  a  U.S.  Army  military  reservation,  in  support  of  AFRL’s  2010  Tech  Warrior  Exercise. 
The  demonstration  involved  the  use  of  the  AFRL-developed,  unmanned,  firefighting  vehicle 
(UFV)  known  as  the  “Fire  Defender”  to  showcase  a  first  generation  of  autonomous  UFV 
behaviors  to  assist  emergency  first  responders/firefighters  in  hazardous  fire  environments. 

AFRL/RXQE  approached  the  project  in  three  core  phases.  Phase  1  and  2  involved  developing  a 
teleoperated  proof-of-concept  UFV  by  implementing  first  a  hand-controller-operated  via  line-of- 
sight  (LOS)  engineering  model  (phase  1),  and  then  an  operational  control  unit  (OCU)  using 
extended  LOS  to  improve  situational  awareness  (SA)  and  remote  capability  (phase  2). 

While  the  first  two  phases  concentrated  solely  on  laying  the  groundwork  by  developing  a 
standard  teleoperated  UFV  system,  the  third  phase  focused  on  expanding  the  UFV’s  capabilities 
by  integrating  first-generation  autonomous  UFV  behaviors.  The  embedded  first-generation 
autonomous  behaviors  provided  the  UFV  the  capability  to  detect,  track  and  extinguish  a  fire  at 
the  push  of  a  button.  RXQE  investigated  the  implementation  of  stereovision  by  integrating  a 
long-wave  infrared  (LWIR)  sensor  pair  for  feedback  input  to  the  autonomous  control  system. 

Teleoperation  capabilities  are  a  common  and  mature  UGS  technology;  however,  integrating 
autonomous  behaviors  into  a  UGS  is  a  difficult,  task-dependent  challenge.  The  key  focus  in  this 
R&D  effort  was  to  develop  a  UFV  with  first- generation  autonomous  firefighting  behaviors. 
RXQE  was  successful  in  demonstrating  a  teleoperable  and  semiautonomous  UFV  suitable  for 
DoD  firefighting  CONOPS.  Both  Tyndall  AFB  and  Ft.  Drum  provided  relevant  test 
environments  to  demonstrate  the  UFV’s  added  remote  firefighting  capabilities  in  an  aircraft  fire 
scenario.  However,  to  improve  the  art  of  autonomous  UFV  firefighting,  additional  sensor 
technologies  and  control  methods  need  to  be  investigated  to  the  overcome  this  challenge.  For 
instance,  wind  in  excess  of  5  mph  was  a  major  problem  that  significantly  affected  the 
perfonnance  and  accuracy  of  the  autonomous  UFV  behaviors,  as  were  blooming  effects  from 
saturated  thermal  reflections. 
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2.  INTRODUCTION 

2.1.  AFRL/RXQE  Team 

Government  Civilians  (AFRL/RXQE):  Lucas  Martinez  (Technical  Lead) 

Contractors:  ARA — David  Trevvett  (Program  Manager/Test  Support),  Wintec — Alex  Evans 
(UFV  operator  support),  Bordie  Casiple  (Test  Support) 

2.2.  Purpose 

The  purpose  of  this  project  was  to  develop  a  UFV  to  demonstrate  remote  capability  for  fighting 
fires  at  safe  distances  via  manual  teleoperation  and  autonomously.  Additionally,  the  first- 
generation  autonomous  features  were  developed  for  the  UFV  to  detect  and  localize  the  position 
of  the  fire  with  respect  to  the  platform,  and  autonomously  apply  fire  extinguishing  agent  to  the 
seat  of  the  fire. 

2.3.  Background 

Firefighters  are  faced  with  battling  day-to-day  common  fires — donns,  base  housing,  etc.,  as  well 
as  the  unique  challenges  of  battling  aircraft  fires.  Any  of  these  types  of  fires  place  firefighters  in 
hazardous  environments,  not  just  from  the  fire  and  heat,  but  from  explosive  materials  such  as 
aircraft  bombs,  and  hazardous  chemicals  off-gassing  such  as  carbon  monoxide  and  “hydrogen 
cyanide  (a  byproduct  of  combustion  of  materials:  insulation,  carpets,  clothing,  and 
synthetics)”^1  .  Consequently,  fire  environments  pose  extreme  life-threatening  risks  to 
firefighters,  which  could  result  in  the  loss  of  human  life. 

Currently,  U.S.  military  UGS  systems  are  often  used  to  provide  protection  against  from 
Chemical,  Biological,  Radiological,  Nuclear  and  high-yield  Explosive  (CBRNE)  threats,  and 
during  Explosive  Ordnance  Disposal  (EOD)  and  Improvised  Explosive  Device  (IED)  mitigation 
missions.  For  the  past  3  years,  AFRL/RXQE  has  been  investigating  ways  to  expand  the 
capabilities  of  UGS  to  perform  these  missions  by  incorporating  a  teleoperated  UFV. 

Due  to  increased  customer  interest  from  the  field  and  the  inherent  dangers  involved  in 
firefighting  operations,  Air  Force  Space  Command  (AFSPC)  in  conjunction  with  Air  Force  Civil 
Engineer  Support  Agency  (AFCESA)  requested  the  development  of  a  proof-of-concept  UFV. 
RXQE  engineers  from  the  fire  research  group  with  experience  in  extinguishing  systems  and 
robotics  group  with  experience  in  developing  UGS  initiated  a  joint  Fire/Robotics  project.  AFRL 
leveraged  the  project  with  the  use  of  RXQE’s  base  robotic  vehicle  platform,  known  as  the 
Defender/Land  Tamer  '121.  The  initial  R&D  prototyping  effort  lasted  only  10  months  and  ended 
with  a  successful  demonstration  at  the  Tyndall  AFB,  FL,  fire  pit  in  an  aircraft  fire  scenario  in  late 
November  2008,  shown  in  Figure  1. 

The  demonstration  of  the  Fire  Defender  UFV  proved  it  to  be  a  safe,  effective,  and  reliable  means 
to  defeat  an  aircraft  fire — Fire  Defender  extinguished  a  JP-8  Class  B  fire  within  less  than  one  and 
a  half  minutes,  applying  a  mixture  of  100  gallons  of  water  and  3  gallons  of  6%  Aqueous  Film 
Fonning  Foam  (AFFF). 
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Figure  1.  Fire  Defender  Demonstration  at  Tyndall  AFB 


2.4.  Scope 

This  project  comprised  three  phases  during  a  3-year  effort.  Phase  1  focused  on  the  initial 
development  of  the  UFV  “proof  of  concept”  by  designing  a  firefighting  payload  and  integrating 
it  into  AFRL’s  Defender  unmanned  ground  vehicle  (UGV)  and  simply  teleoperating  the  UFV 
with  a  wireless  hand  controller  via  LOS.  (Details  of  the  system  are  given  in  Appendix  A.)  Phase 
2  involved  upgrading  the  UFV  by  developing  an  OCU  to  extend  the  LOS  range,  operator 
situational  awareness,  and  controls.  (Details  of  the  upgrade  are  given  in  Appendix  B.)  Lastly, 
Phase  3  consisted  of  developing  first-generation  autonomous  behaviors  to  control  the  turret 
nozzle  to  effectively  apply  fire  extinguishing  agent  onto  the  seat  of  the  fire. 

2.5.  Goals  and  Objectives 

This  project  sought  to  advance  the  current  state  of  technology  available  to  DoD  firefighters  and 
emergency  first  responders  by  developing  a  UFV  prototype  able  to  extinguish  small  Class  A, 
ordinary  combustibles,  fuel  fires.  AFRL’s  major  goal  was  to  deliver  a  working  UFV  prototype 
with  manual  teleoperation  and  autonomous  control  capabilities.  To  achieve  this,  AFRL/RXQE 
led  the  R&D  effort  and  leveraged  the  existing  teleoperated  Defender  UGV  platform.  Below  is  a 
list  of  the  key  goals  and  objectives  of  the  project: 
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Goals 

•  Develop  a  basic  teleoperated 
UGV  with  firefighting  payload. 

•  Develop  an  OCU  for  the  UFV. 

•  Develop  first-generation 
autonomous  UFV  firefighting 
behaviors. 

•  Demonstrate  and  evaluate  both 
manual  teleoperated  and 
autonomous  UFV  capabilities. 


RXQE  demonstrated  and  evaluated  first-generation  autonomous  firefighting  behaviors  in  a 
relevant  testing  environment  during  AFRL’s  2010  Tech  Warrior  exercise  at  Ft.  Drum’s 
Firefighting  Training  Center.  Tech  Warrior  provided  the  opportunity  to  test  the  robotic  platform 
with  the  basic  autonomous  behaviors  and  robotic  firefighting  UFV  technologies  in  the  presence 
of  fire  fighters,  potential  future  end  users.  In  addition,  the  firefighters  were  able  to  provide  the 
AFRL  Robotics  Team  valuable  feedback  on  the  operator/user  interface  and  controls,  mobility, 
and  firefighting  perfonnance.  The  collection  of  this  input  is  beneficial  to  the  future  development 
of  robotic  firefighting  platfonns. 

The  implementation  of  a  robotic  firefighting  vehicle  is  expected  to  significantly  improve  the 
safety  and  operational  efficiency  of  emergency  first  responders  by  providing  a  safe  stand-off 
distance  from  a  remote  position  to  mitigate  such  dangerous  firefighting  environments  as  space 
launch  complexes,  aircraft  fires,  and  munitions  and  hypergolic  fuel  storage  facilities,  where  a 
multitude  of  volatile  fire  hazards  present  a  potential  loss  of  life. 


Obiectives 

•  Demonstrate  a  proof-of-concept  UFV  to  enhance 
DoD  firefighters’  mission  capability. 

•  Extend  the  teleoperation  LOS  range  and  provide 
better  situational  awareness  to  the  end  user 
(firefighter). 

•  Expand  the  operational  efficiency  by  automating 
firefighting  methods. 

•  Demonstrate  a  UFV  concept  to  the  firefighting 
community  to  provide  awareness  of  the  current 
technology  capabilities,  to  gain  firefighter  end- 
user  acceptance,  and  to  get  operational  feedback. 


4 

Distribution  A:  Approved  for  public  release;  distribution  unlimited.  88ABW-2012-2144,  12  April  2012. 


Robotic  Firefighting  Autonomous  Technologies  Report 


3.  METHODS,  ASSUMPTIONS  AND  PROCEDURES 

With  ever-evolving  advances  in  the  level  of  autonomy  integrated  into  UGVs,  RXQE  confronted 
the  tough  challenge  of  developing  first-generation  autonomous  firefighting  behaviors  to  extend 
the  capabilities  and  increase  the  efficiency  of  unmanned  firefighting  techniques,  tactics  and 
procedures  (TTPs). 

The  R&D  effort  for  phase  III  of  the  project  began  in  2010.  The  goal  was  to  demonstrate  that  the 
Fire  Defender  UFV  could  accurately  detect  and  detennine  the  position  and  distance  of  the  hot 
spot  of  a  heat  source  (with  respect  to  the  UFV),  autonomously  direct  the  Fire  Defender’s  turret  to 
apply  firefighting  agent  (water  and  foam  mix)  onto  the  hot  spot  and  begin  a  sweeping  pattern 
over  the  region  of  the  heat/fuel  source.  This  effort  would  be  a  first  step  into  the  world  of 
unmanned  firefighting  autonomous  behaviors,  thus  creating  an  innovative  viewpoint  from  the 
traditional  ways  firefighters  perfonn  and  execute  their  missions. 

3.1.  Methodology 

In  developing  the  first-generation  UFV  autonomous  behaviors,  computer  stereovision  technology 
was  chosen  as  the  primary  sensor  for  the  UFV.  Computer  stereovision  is  an  advanced  sensor 
used  in  robotics  to  create  accurate  depth  perception  and  image  localization  of  objects  in  three 
dimensional  (3-D)  space  for  obstacle  detection  and  range  finding.  This  technique  is  used  to 
exploit  the  visual  incongruity  caused  by  the  object’s  parallax  when  seen  from  one  camera 
viewpoint  versus  another  camera  viewpoint. 

Two  cameras  are  used  in  stereovision  applications  to  provide  accurate  visual  perception  and  3-D 
spatial  detail.  This  is  accomplished  by  a  computer  processing  technique  that  compares  the 
images,  while  shifting  the  two  images  so  they  are  superimposed  into  one  complete  image.  The 
amount  the  image  is  shifted  is  known  as  disparity.  The  disparity  is  then  used  to  compute  the 
distance  to  the  object  of  interest. 

For  this  application,  the  stereovision  system  comprised  two  Bullard®  LWIR  imagers  to  provide  a 
centralized  location  of  the  heat  source/fire.  This  unique  configuration  relays  the  distance  and 
relative  position  data  of  the  fire  with  respect  to  the  UFV  in  real  time.  To  correctly  apply  the 
firefighting  agent  onto  the  location  of  the  seat  (hotspot)  of  the  fire,  accurate  feedback  must  be 
provided  to  command  the  turret’s  nozzle  position. 

To  calculate  the  disparity  of  the  object  and  the  local  position  of  the  heat  source  the  object  of 
interest  must  be  identified  on  both  camera  views.  This  task  was  simplified  by  the  use  of  the 
Bullard®  LWIR  imagers,  because  these  imagers  have  a  mode  that  color  codes  the  pixels  in 
proportion  to  relative  temperature.  Using  these  data,  together  with  baseline  information  and 
calibration  data,  it  was  possible  to  calculate  the  image  disparity.  The  target  depth  is  inversely 
proportional  to  the  disparity  measure  and  was  used  to  fully  define  the  local  position  of  the  heat 
source  with  respect  to  the  cameras. 

Figure  2  illustrates  the  theory  of  visual  perception  and  shows  how  stereovision  is  interpreted  by 
the  computer’s  machine-visual  interface.  Camera  views  overlap  in  the  green  area  and  the 
computer  merges  the  two  views  into  one  image.  This  region  corresponding  to  the  field  of  view  of 
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Figure  2.  Depiction  of  Fire  Defender  Stereovision 


the  UFV’s  sensors  is  known  as  the  horopter.  The  horopter  contains  all  the  visual  information  that 
provides  the  distance  feedback  from  the  heat  source  to  the  control  system.  Any  objects  of  interest 
outside  this  field-of-view  region  cannot  be  identified  with  the  implementation  of  stereovision. 

The  two  bottom  frames  of  Figure  3  are  the  left  and  right  LWIR  camera  views,  respectively,  of 
the  Fire  Defender  UFV.  The  blue  region  corresponds  to  the  thermal  region  sensed  by  the  LWIR. 
The  images  are  digitally  processed  in  near-real  time  to  calculate  the  centralized  3-D  relative 
location  with  the  azimuth,  elevation,  and  distance  of  the  hotspot  of  the  fire.  The  two  top  frames 
are  electrical-optical  (EO)  camera  views  use  to  provide  a  visual  perception  of  the  front  and  turret 
view  respectively,  from  the  UFV  during  all  modes  of  operation. 


Figure  3.  Visual  and  LWIR  Left  and  Right  Images 
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3.2.  Technical  Design  Approach 

To  develop  the  first-generation  autonomous  control  behaviors  for  the  Fire  Defender  UFV,  two 
LWIR  cameras  were  mounted  symmetrically  about  the  center  line  of  the  vehicle  with  respect  to 
the  location  of  the  turret  to  the  existing  UFV  configuration.  This  creates  the  stereovision  effect 
and  provides  enhanced  visual  LWIR  data  for  the  autonomous  control  system  to  acquire  accurate 
hotspot  distance  feedback  and  control  the  turret’s  azimuth  and  pitch  position. 

3.2.1.  System  Design  Modification 

Figure  4  highlights  some  of  the  key  essential  components  of  the  UFV  system  with  the  addition  of 
the  Bullard "  LWIR  cameras  denoted  by  the  number  (4)  label.  These  sensors  provide  the  UFV 
with  the  stereovision  data  as  control  feedback  to  pinpoint  the  location  of  the  hotspot  of  the  fire. 
The  electronically  controlled  turret  (1)  has  a  180°  pan/tilt  control  function  to  allow  the  turret  full 
range  of  motion  to  direct  the  firefighting  agent.  With  the  digital  infrared  imager  (2)  and  standard 
EO  camera  (3)  the  firefighter  (end  user)  has  the  option  to  observe  views  from  either  the  front  or 
behind  the  turret  on  the  OCU. 


(1 )  Electronically  Controlled  Turret  (Pan  &  l  ilt)  &  No77le  (straight  and  dispersed  streams) 

(2)  Digital  Infrared  Imager  -  DI-7000  (Pan/Tilt  EO  and  IR  camera  combination) 
Firefighting  System  Turret  View  (as  shown  in  Figure  6  top  right  frame) 

(3)  Standard  EO  Bullet  Camera 

(4)  Bullard®  LWIR  Camera 

(5)  Hydraulic  Centrifugal  Water  Pump 

(6)  Electronic  Controlled  Valves  (High  Mass  Flow) 

Figure  4.  Fire  Defender  Autonomous  Configuration 
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Figure  5  illustrates  the  modified  version  of  the  teleoperated  Fire  Defender  controller  configured 
with  the  enhanced  autonomous  mode  and  control  features.  This  allows  the  firefighter  to  control 
the  UFV  in  both  teleoperation  and  autonomous  modes.  The  left  joystick  (10)  controls  the 
mobility  of  the  UFV,  while  the  right  joystick  (7)  controls  the  pan/tilt  function  of  the  turret.  The 
controller  is  a  plug  and  play  device  that  connects  into  the  OCU  thru  the  universal  serial  bus 
(USB).  In  addition,  the  firefighter  can  switch  through  the  various  cameras  views  (9)  such  as  the 
turret  and  front  view.  To  engage  in  autonomous  firefighting,  the  firefighter  just  switches  to 
autonomous  mode  (1)  and  follows  the  sequence  as  instructed  in  Appendix  C. 


XAW  fSi  (BKc 


“V 


VIEW 

SELECT 


( 1 )  Manual/Autonomous  Mode 

(2)  Nozzle  Valve  Switch 

(3)  Tank  Valve  Switch 

(4)  Hydraulic  Water  Pump  Switch 

(5)  Autonomous  Turret  Position 
Switch 

(6)  Autonomous  Programmable 
Fixed  Pattern  Switch 

(7)  Manual  Turret  (Pan/Tilt) 

Control 

(8)  Nozzle  (straight,  and  dispersed 
stream)  Control 

(9)  Camera  View  Select 

(10)  UFV  Vehicle  Dynamic  Joystick 
Controller 

(11)  Engine  RPM  Control  -  Regulates 
Water  Output  Pressure 


Figure  5.  Fire  Defender  Controller 


3.3.  System  Characterization 

To  achieve  the  challenging  task  of  incorporating  autonomous  firefighting  behaviors  into  the  Fire 
Defender,  two  principal  design  objectives  were  investigated.  The  first  and  core  objective  was  to 
find  a  thermal  imaging  sensor  to  identify  and  sufficiently  localize  the  3-D  relative  position  of  a 
hot  spot  region  of  a  fire.  This  data  feedback  serves  as  valuable  input  to  the  control  system  to  be 
able  to  direct  the  appropriate  position  (azimuth  and  pitch  angle)  of  the  turret. 

The  second  objective  was  to  acquire  data  to  characterize  the  effective  discharge  distance  of  the 
firefighting  turret  nozzle  with  respect  to  its  various  pitch  angles.  The  combination  of  both  the 
distance  feedback  and  turret  perfonnance  parameters  enabled  the  UFV’s  firefighting  payload  to 
be  automated. 
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4.  RESULTS  AND  DISCUSSION 
4.1.  IR  Calibration 

To  evaluate  the  effectiveness  of  implementing  a  stereovision  application  with  the  LWIR 
Bullard  ®  cameras  and  to  provide  the  3-D  relative  position  of  the  seat  of  a  fire,  a  test  bed  was  set 
up  consisting  of  a  heat  source  (70  lbs  of  charcoal  in  a  metal  container)  set  at  50-ft  intervals  from 
50  ft  to  300  ft  along  the  central  axis  of  the  Fire  Defender.  Data  were  recorded  using  the  LWIR 
sensors  at  each  incremental  location  for  the  computed  distance  of  the  hotspot  region  and 
compared  to  the  actual  distance. 

Data  revealed  in  Figure  6  illustrate  that  the  computed  distance  from  the  hot  spot  became  more 
accurate  as  the  Fire  Defender  approached  closer  to  the  heat  source.  At  200  ft  and  closer  the 
computed  distance  became  increasingly  more  accurate  and  at  the  effective  firefighting  range  of 
50  ft  the  computed  distance  was  within  5  ft  tolerance  of  the  actual  distance. 

The  red  and  blue  lines,  respectively,  in  Figure  6  represent  maximum  and  minimum  distance 
computations  by  the  stereovision  software  algorithm  from  five  data  sets  at  each  50-ft  increment 
ranging  from  50  ft  to  300  ft. 


-❖-Minimum  Calculated  Value  “•“Maximum  Calculated  Value  -o- Actual  Distance  of  Heat  Source 


0  25  50  75  100  125  150  175  200  225  250  275  300  325 

Distance  in  Feet  (Ft.) 

Figure  6.  LWIR  Stereo  vision  Heat  Source  Distance  Plot 


The  red  cursor  shown  in  Figure  7  identifies  the  centroid  of  the  heat  source  to  determine  the 
distance,  pitch  and  azimuth.  At  300  ft  the  data  were  obscured  due  to  the  limitations  of  the  LWIR 
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Figure  7.  Left  LWIR  Camera  Outputs  at  300  ft 


camera  sensors.  The  gains  of  the  sensors  had  to  be  adjusted  to  a  high  level  to  discriminate  the 
heat  source  from  heat  reflecting  from  the  pavement,  which  interfered  with  acquiring  accurate 
thermal  data.  The  real-time  captured  images  of  the  left  camera  video  feed  in  Figure  7  show  the 
instability  of  the  distance  data  feedback.  The  red  cursor  would  vary  back  and  forth  consistently 
between  the  circled  positions  at  around  300  ft,  which  corrupted  the  distance  feedback  data.  Due 
to  the  erratic  behavior  of  the  red  cursor  at  300  ft,  the  stereovision  software  algorithm  was  unable 
to  determine  the  distance  accurately.  However,  as  the  distance  neared  200  ft  the  accuracy  of  the 
computed  distance  improved  significantly. 

At  50  ft,  the  results  improved  dramatically  and  the  red  cursors  consistently  provided  valid 
computed  distance  feedback  with  an  accuracy  of  5  ft.  Figure  8  illustrates  the  stability  and 
accuracy  of  the  red  cross-hairs  pinpointing  the  hotspot  region  from  both  the  left  and  the  right 
LWIR  camera. 


Figure  8.  Left  and  Right  LWIR  Camera  Outputs  at  50  ft 
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4.2.  Modeling  Effective  Range  of  Agent  Application 

To  characterize  the  UFV’s  effective  range,  the  turret  was  set  at  different  pitch  angles  beginning 
with  the  horizontal  limit  of  0°.  The  angle  of  pitch  was  adjusted  incrementally  around  5°  to 
achieve  approximately  10-ft  increases  in  discharge  range.  It  was  found  that  the  longest  effective 
range  of  the  firefighting  system  is  60  ft,  but  this  varies  depending  on  the  direction  of  head,  tail 
and  cross  winds. 

Figure  9  contains  collected  data  points  characterizing  the  UFV’s  effective  range.  The  second  data 
point  recorded  in  data  set  2,  where  the  pitch  angle  was  zero  and  the  trajectory  remained  at  50  ft, 
does  not  agree  with  the  second  data  points  recorded  from  data  sets  1  and  3.  This  was  a  result  of 
inconsistent  behavior  of  the  Fire  Defender’s  vehicle  hydraulic  flow  supplied  to  the  firefighting 
system’s  hydraulic  centrifugal  pump.  For  all  three  cases,  it  was  discovered  that  the  stream  breaks 
apart  by  the  time  it  reaches  70  ft.  The  water  stream  is  very  sensitive  to  wind  affecting  both  its 
range  and  lateral  accuracy.  These  data  points  reflect  ideal  conditions,  under  which  the  wind  is 
not  a  significant  factor. 


Data  Set  1  ^KDataSet2  -A-DataSetS 


Figure  9.  Turret/Nozzle  Effective  Discharge  Range  Plot 


4.3.  Field  Evaluation  of  UFV  Teleoperation  and  Autonomous  Capabilities 

In  July  2010,  RXQE  personnel  participated  in  AFRL’s  Tech  Warrior  exercise  at  Ft.  Drum,  NY. 
Tech  Warrior  served  as  a  venue  for  AFRL  scientists  and  engineers  to  conduct  field  evaluations  in 
a  relevant  test  environment  with  access  to  a  mock-up  aircraft,  a  jet  fuel  pit,  and  a  vehicle-bome 
IED  scenario.  AFRL  personnel  provided  a  hands-on  demonstration  of  the  Fire  Defender  UFV  to 
military  and  civilian  firefighters  (the  end  users),  and  interfaced  with  them  to  acquire  operational 
feedback  of  the  UFV’s  performance. 
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4.3.1.  Tech  Warrior  Exercise  Day  1 

The  RXQE  team  arrived  at  the  Ft.  Drum  Fire  Training  Center  to  configure  equipment  and 
prepare  for  test  experiments.  In  preparation  the  following  was  performed: 

•  Operational  testing  validated  that  the  communication  system,  OCU,  UFV  and  firefighting 
payload  were  functioning  properly. 

•  Class  A  ordinary  combustible  fuels  were  used  to  ignite  test  fires  that  produced  an  800  °F 
heat  source.  These  fires  were  used  to  calibrate  the  Bullard8  LWIR  cameras  used  for  fire 
detection  at  measured  distances  out  to  100  ft  in  20-ft  increments. 

•  Agent  throw  tests  evaluated  the  effective  discharge  distance  of  the  firefighting  agent. 

The  test  equipment  was  configured  and  operationally  tested,  and  no  performance  or  functionality 
problems  were  encountered.  The  team  established  test  parameters  and  configured  a  test  bed  for 
the  experiments.  Preliminary  IR  camera  calibration  and  effective  throw  distance  testing  of  the 
firefighting  agent  was  conducted  in  preparation  for  tests.  The  most  effective  throw  distance  was 
between  60  and  75  ft  with  a  total  reach  distance  at  100  ft.  Between  65  and  100  ft  the  stream  was 
not  concentrated  and  dispersed  greatly.  The  IR  cameras  were  effective  in  determining  the 
location  of  the  heat  source  at  a  distance  up  to  100  ft  with  a  tolerance  of  ±  5  ft. 

4.3.2.  Tech  Warrior  Exercise  Day  2 

A  series  of  experiments  was  conducted  on  fire  detection  and  autonomous  extinguishment 
software  and  behaviors. 

Tests  1-3: 

•  Using  the  test  bed  constructed  on  day  one,  the  team  established  a  Class  A  heat  source 
(450-800  °F)  utilizing  an  aircraft  mock-up  to  simulate  an  aircraft  incident  (the  fire  was 
located  in  the  overwing  emergency  exit).  Infrared  (IR)  camera  settings  were  reconfirmed 
and  weather  readings  were  recorded  (winds  out  of  the  south-southeast  at  6  mph,  74  °F) 

•  The  UFV  was  positioned  at  a  stand-off  distance  of  approximately  200  yd  from  the 
simulated  aircraft  fire  and  teleoperated  in  incremental  distances  toward  the  aircraft  fire 
until  it  was  within  effective  firefighting  range. 

•  The  IR  system  recognized  the  heat  source  and  positioned  the  turret  in  the  direction  of  the 
fire.  The  system  automatically  compensated  for  distance  (elevation  in  relationship  to 
distance)  and  was  placed  into  autonomous  firefighting  mode. 

•  Firefighting  agent  was  discharged  from  a  distance  of  60  ft  in  a  sweeping  “Z”  pattern  to 
maximize  likelihood  of  effective  application. 

•  LWIR  camera  gains,  effective  firefighting  range  and  autonomous  behaviors  were 
recorded  via  an  on-board  digital  recorder  and  stand-alone  video  recorders.  Adjustments  to 
the  system  were  made  to  optimize  extinguishing  agent  effectiveness  and  tests  were 
repeated  to  validate  previous  test  findings  and  system  recalibrations. 

Tests  4-5: 

•  Constructed  a  new  test  bed  using  the  same  graduated  scale  as  test  1 .  The  heat  source  was 
moved  into  the  open  to  simulate  a  fire  under  the  aircraft  wing. 

•  The  cameras  were  recalibrated  and  throw  distances  confirmed  prior  to  the  first  underwing 
(open  area)  fire. 
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•  The  team  followed  the  same  protocols  as  above  and,  as  indicated  in  the  test  plan,  two 
more  tests  were  completed. 

•  During  the  series  of  ground  tests  a  marked  difference  in  the  autonomous  elevation 
compensation  for  distance  (resulting  in  farther  overall  throw  distance)  was  discovered. 
This  was  attributed  to  the  nozzle  adjustment,  fire  location  and  weather  conditions 
(sporadic  and  prevailing  winds).  Further  testing  and  data  collection  were  both  repeatable 
and  reliable. 

A  total  of  five  calibrated  tests  were  conducted  utilizing  an  ordinary  Class  A  combustible  heat 
source  and  an  aircraft  mock-up  to  simulate  an  aviation  accident.  The  UFV  performed  without 
failure  and  met  expectations  for  autonomous  fire  detection  and  extinguishing  behaviors.  When 
calibrated  to  a  given  set  of  fire  behavior  parameters,  early  flame  recognition  and  fire 
extinguishing  behaviors  were  replicated.  When  the  dynamics  of  the  experiment  were  changed, 
the  system  was  recalibrated  and  repeatable  results  were  obtained.  Data  were  recorded  to  validate 
the  changes. 

Due  to  the  successful  experiments  during  day  two,  the  number  of  required  experiments  was 
completed  and  it  was  decided  that  advanced,  large-scale  testing  was  needed  to  better  evaluate  the 
system.  The  on-site  support  contact,  Assistant  Chief  William  Maciorowski,  was  contacted  and  he 
volunteered  the  use  of  the  Fort  Drum  Class  B  fire  pit  and  enough  JP-8  aviation  fuel  to  conduct 
two  medium-scale  pool  fires.  He  also  made  an  automobile  available  for  use  as  a  prop  to  simulate 
an  IED  explosion  and  fire.  The  remainder  of  the  day  was  spent  making  preparations  for  advanced 
testing  with  Ft.  Drum  firefighting  personnel. 

4.3.3.  Tech  Warrior  Exercise  Day  3 

The  third  day  at  the  Ft.  Drum  Fire  Training  Center  was  used  to  conduct  a  series  of  experiments 
with  the  support  of  the  Ft.  Drum  Fire  Department.  The  experiments  consisted  of  two  JP-8 
aviation  Class  B  fires  to  simulate  an  aircraft  fire,  and  a  Class  A  fire  in  a  car  mock-up  simulating 
an  IED  explosion  and  fire.  The  Ft.  Drum  firefighters  controlled  the  UFV  via  teleoperation  with 
the  first-generation  autonomous  capability  for  these  experiments. 

Test  5  End  User’s  Hands-on  Demonstration 

The  Ft.  Drum  Fire  Department  created  two,  sequential,  medium-scale  pool  fires  at  their  Class  B 
fire  pit  with  ~50  lbs  of  JP-8.  During  these  two  tests,  a  firefighter  was  used  as  a  test  subject  to 
evaluate  the  user-friendliness  of  the  controls  and  to  provide  end-user  feedback  about  the  whole 
system  in  use.  As  part  of  the  exercise,  the  firefighter  fully  controlled  the  UFV  and  extinguished 
the  JP-8  fuel  fire  via  teleoperation  after  activating  the  autonomous  firefighting  mode. 

Test  6  Vehicle-borne  IED  Mock  up 

The  Ft.  Drum  Fire  Department  set  a  Class  A  fire  inside  a  car  mock-up  to  simulate  a  car  bombing 
fire.  During  this  test,  the  RXQE  team  extinguished  the  fire  and  collected  more  thermal  and  video 
data. 

Summary 

The  Ft.  Drum  Fire  Department  provided  two  additional  advanced  tests  using  their  training 
facility  and  crew.  This  support  enabled  the  gathering  of  valuable  data  along  with  additional  user 
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feedback.  The  firefighters  reported  that  having  a  firefighting  robotic  asset  would  be  of  great  use 
in  real-world  situations  that  include  dangerous  HazMat  conditions.  During  conversations  with 
firefighting  personnel,  many  of  them  expressed  great  interest  in  using  such  an  asset  at  the  Ft. 
Drum  Anny  Post,  where  active  range  fires  containing  unexploded  ordnance  (UXO)  occur  quite 
frequently  and  pose  extreme  risks  during  firefighting  missions. 

4.4.  Key  Observations  and  Takeaways 

As  with  any  R&D  effort  in  the  early  phases  of  development  there  were  design  challenges, 
concept  refinements  and  considerations  that  need  to  be  addressed  and  evaluated  before  entering 
the  next  level  of  research  and  development.  To  date,  some  key  takeaways  and  issues  associated 
with  current  R&D  work  include  appropriate  vehicle  selection,  firefighting  performance,  capacity, 
payload  size,  selection  of  a  thermal  imager  sensor,  refinements  to  the  autonomous  behavior 
software  algorithm,  and  understanding  how  the  use  of  a  UFV  will  fit  into  the  CONOPS  of 
today’s  robotic  firefighting. 

The  current  Fire  Defender  vehicle  platfonn,  the  Land  Tamer®,  served  as  a  cost-effective 
prototype  system  for  a  proof-of-concept  demonstration  to  exhibit  robotic  firefighting  applications 
and  autonomous  behaviors.  This  UFV  system  is  solely  a  proof  of  concept,  which  would  not  be 
fielded  in  its  current  form  due  to  limitations  in  payload  capacity,  terrain  mobility  and 
perfonnance  capabilities.  The  next  R&D  iteration  envisioned  is  to  transfer  the  technology  with 
improvements  in  both  autonomy  and  terrain  capabilities  by  implementing  a  large-scale  tracked 
vehicle  and  adding  an  ultrahigh-pressure  firefighting  payload  with  a  higher-capacity  water  tank. 

Thermal  imaging  sensing  is  a  critical  component  of  the  autonomous  control  system.  For  this 
application,  two  LWIR  imagers  were  used  in  a  stereovision  arrangement.  The  selected  LWIR 
imagers  were  successful  in  providing  adequate  feedback,  but  there  were  many  limitations  of  the 
sensors’  configured  settings  and  features.  Different  intensities  of  fire  required  manual  adjustment 
of  the  sensors’  gains,  meaning  a  person  would  have  to  make  those  adjustments  on  board  the 
UFV.  This  is  not  practical  in  an  unmanned  firefighting  scenario  during  the  remote  operation  of  a 
UFV. 

It  was  discovered  that  large  heat  sources  saturated  the  sensors  because  of  blooming  effects  and 
reflections.  This  rendered  the  autonomous  control  system’s  algorithms  useless  in  localizing  the 
position  of  the  heat  source/fire  without  manual  adjustments.  The  future  sensor  must  have  the 
ability  to  self-calibrate  and  include  a  range  of  gains  appropriate  to  compensate  for  unpredictable 
fire  sizes  and  temperature  gradients. 

Wind  plays  a  significant  role  in  the  performance  of  the  UFV  system.  Depending  on  the  strength 
and  velocity  of  head,  tail  or  crosswinds,  the  dispersal  application  of  the  firefighting  agent  is 
unpredictable  and  imprecise.  The  current  agent  trajectories  of  firefighting  system  were 
characterized  under  ideal  weather  conditions  where  wind  did  not  affect  the  parameters.  Thus, 
further  research  needs  to  address  compensating  for  this  problem  when  implementing  the 
autonomous  solution/behaviors  during  adverse  weather/wind  conditions. 
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5.  CONCLUSIONS 

Robotic  firefighting  is  an  innovative  concept  that  may  significantly  benefit  firefighters  with 
enhanced  capabilities,  operational  safety  and  efficiency.  Unmanned  firefighting  systems  do  not 
exist  in  the  theater  and  are  not  in  the  hands  of  everyday  firefighters.  The  fire  community  lacks 
awareness  of  the  advantages  of  the  UFV  system  technologies  and  its  force  multiplying 
capability.  With  a  better  understanding  of  firefighting  mission  requirements  and  CONOPS, 
unmanned  firefighting  systems  can  be  designed  to  meet  those  needs. 

Further  research  efforts  to  improve  upon  the  current  state  of  the  art  should  focus  on  improving 
firefighting  automated  tasks.  From  an  operational  standpoint,  it  would  be  more  effective  for  the 
end  user  to  teleoperate  the  UFV  in  autonomous  firefighting  mode,  so  control  of  the  firefighting 
system  is  hands-free  and  the  operator’s  only  focus  is  on  controlling  the  vehicle.  A  more  suitable 
LWIR  sensor  that  can  self-calibrate  to  the  thennal  response  to  avoid  sensor  saturation  effects  and 
discriminate  thennal  reflections  in  sensor  feedback  is  required.  Self-adaptive  control  algorithms 
need  to  be  enhanced  to  compensate  for  factors  such  as  wind,  to  differentiate  between  the  actual 
fire  and  thennal  reflection,  and  to  distinguish  multiple  fires — as  in  the  case  of  two  isolated  fires 
close  to  one  another.  The  algorithm  should  be  able  to  autonomously  identify  and  engage  them  as 
separate  fires,  perhaps  dependent  on  a  priority  command  issued  by  the  firefighter  (end  user). 
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Appendix  A:  Fire  Defender  Initial  Prototype 

The  Fire  Defender  (Fig.  A-l)  is  a  Land  Tamer®  UGV  fitted  with  a  modular,  self-contained  200- 
gal  firefighting  system/payload.  The  teleoperated  UFV  is  controlled  through  a  2.4-GHz  RF  link 
by  a  portable  customized  Miratron®  controller,  which  requires  that  the  end  user  must  maintain 
line-of-sight  (LOS)  contact  and  a  physical  visual  of  the  UFV  during  operations.  This  allows  the 
end  user  to  be  located  at  a  relatively  safe  distance  with  an  effective  operating  range  up  to  100  m. 


AFRL  RXOEs  Defender  UGV 


Figure  A-l.  Teleoperated  Fire  Defender  First-generation  UFV  Prototype 


The  controller  for  the  initial  UFV  prototype  was  custom  designed  for  the  application.  The  left 
half  of  the  unit  controls  the  vehicle  and  the  right  half  controls  the  functions  of  the  firefighting 
payload,  as  seen  in  Figure  A-2.  The  controls  are  supposed  to  be  intuitive,  allowing  the  firefighter 
to  pick  up  the  controller  and  operate  the  UFV  in  real-life  situations  with  rapid  response  and 
engagement.  Also,  the  controller  is  specifically  designed  to  strap  to  the  firefighter’s  waist  line; 
the  ergonomic  fit  provides  comfort  to  the  operator  during  use  of  the  Fire  Defender  UFV. 


17 

Distribution  A:  Approved  for  public  release;  distribution  unlimited.  88ABW-2012-2144,  12  April  2012. 


Robotic  Firefighting  Autonomous  Technologies  Report 


UFV  Vehicle  Control  Firefighting  Payload  Control 


Figure  A-2.  Fire  Defender  Miratron  Controller 


Fire  Defender  Initial  Prototype  System  Description 

The  first-generation  prototype  of  the  Fire  Defender  consists  of  the  following  platforms  and 
specifications: 

•  Land  Tamer1'  UGV  Platfonn 

o  6X6  All-terrain  Vehicle  (ATV) 
o  60-hp  Kubota1'  Turbo  Diesel 

o  Modular  Payload  Deck  for  Mission-Specific  Applications 

■  Force  Protection 

■  Intelligence,  Surveillance  &  Reconnaissance  (ISR) 

■  Firefighting 
o  17-gal  Fuel  Capacity 

o  25-30  mph  Max  Speeds 

•  Firefighting  Payload  System 

o  200-gal  Agent  Tank 

o  Hydraulically  Powered  Centrifugal  Water  Pump 

■  60gpm@180psi 

■  3  min  of  Firefighting  Time 
o  Ambient  Temperature  Sensor 

o  Electronically  Controlled  Turret/Nozzle 

■  Straight/Stream/Fog  Adjustable  Settings 

o  User  Intuitive/Friendly  Miratron"  Joystick  Controller 
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Appendix  B:  Second-generation  Fire  Defender 

As  a  result  of  the  success  of  the  initial  R&D  effort,  in  2009  the  AFRL  Robotics  Team  expanded 
the  Fire  Defender’s  capabilities  by  extending  the  remote  operating  range  to  2-3  mi  LOS.  The 
range  was  extended  by  incorporating  a  base  station  equipped  with  an  OCU.  Addition  of  the  OCU 
allows  the  end  user  better  situational  awareness  from  onboard  cameras  displaying  front,  rear  and 
turret  optional  views  from  the  UFV,  and  it  carries  the  additional  advantage  of  separating  the 
operator  farther  from  the  threat.  Figure  B-l  illustrates  the  enhanced  controller  and,  on  the  OCU 
display,  improvements  to  the  UFV  command  and  control  scheme,  vehicle  data  feedback  (RPM, 
fuel  level,  etc.)  and  visual  perception  for  the  end  user. 


OCU  Display  OCU  Controller 


Figure  B-l.  Upgraded  Fire  Defender  Operator  Control  Unit  (OCU)  Controller  and  Display 


OCU  Features 

The  OCU  display  consists  of  the  following  features: 

•  Vehicle  Feedback  (Located  on  the  upper  left  corner  of  the  OCU  display) 

o  Fuel,  Engine  RPM/Temperature,  and  Battery  Levels 
o  Heading  (Compass) 

o  Three  Selectable  Cameras  (Views  from  the  Front  and  Rear  of  the  Vehicle,  and 
from  the  Turret) 

o  Geo-referenced  Map  Overlay 

•  Firefighting  System  Feedback  (Located  on  the  upper  right  corner  of  the  OCU  display) 

o  Tank  Level 
o  Nozzle  Pressure 
o  Ambient  Air  Temperature 
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Appendix  C:  UFV  Controller  Procedure 

The  sequential  steps  and  modes  of  operation  to  operate  the  UFV  with  the  autonomous  behaviors 
are  as  follows  (referenced  to  Fig.  C-l): 


•win* 


VIEW 

SELECT 


-*S-  HEADING 


|  ( 1 )  Manual/Autonomous  Mode 

(2)  Nozzle  Valve  Switch 

(3)  Tank  Valve  Switch 

(4)  Hydraulic  Water  Pump  Switch 

(5)  Autonomous  Turret  Position 
Switch 

(6)  Autonomous  Programmable 
Fixed  Pattern  Switch 

(7)  Manual  Turret  (Pan/Tilt) 

Control 

(8)  Nozzle  (straight,  and  dispersed 
stream)  Control 

(9)  Camera  View  Select 

(10)  UFV  Vehicle  Dynamic  Joystick 
Controller 

(11)  Engine  RPM  Control  -  Regulates 
Water  Output  Pressure 


Figure  C-l.  Fire  Defender  Controller 


•  To  implement  teleoperated  control  use  the  UFV  vehicle  joystick  controller  (10)  to  guide  the 
Fire  Defender  to  the  effective  range  before  engaging  the  autonomous  firefighting  system. 

•  Adjust  the  engine  rpms  with  knob  ( 1 1)  for  adequate  water  output  pressure  and  flow  for  the 
firefighting  system. 

•  Turn  the  Manual/ Autonomous  Switch  (1)  to  autonomous  mode 

•  Turn  on  the  Nozzle  (2)  and  Tank  Valve  (3)  switches  to  enable  water  flow  to  the  firefighting 
system 

•  Turn  on  the  Autonomous  Turret  Position  switch  (5)  to  direct  the  firefighting  Turret/Nozzle  to 
slew  to  the  central  region  of  the  fire. 

•  Turn  on  the  Hydraulic  Water  Pump  switch  (4)  to  allow  water  to  discharge  from  the 
firefighting  system 

•  Turn  on  the  Autonomous  Programmable  Fixed  Pattern  switch  (6)  to  autonomously  engage  in 
the  firelight.  This  feature  controls  the  turret  to  cyclically  repeat  a  figure  eight  pattern  about 
the  center  of  the  hotspot  region  of  the  fire  until  disengaged.  This  pattern  can  be  adjusted  to 
various  limits. 
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LIST  OF  SYMBOLS,  ABBREVIATIONS,  AND  ACRONYMS 

AFRL  Air  Force  Research  Laboratory 

CONOPS  concept  of  operations 

DoD  Department  of  Defense 

EO  electro-optical  (camera) 

ft  feet 

IED  improvised  explosive  device 

IR  infrared 

JP-8  jet  propellant  8 

lbs  pounds 

LOS  line-of-site 

LWIR  long  wave  infrared 

mph  miles  per  hour 

OCU  operation  control  unit 

R&D  research  and  development 

RXQ  Materials  &  Manufacturing  Directorate,  Airbase  Technology  Division 

RXQE  Airbase  Engineering  Development  Branch 

3-D  three  dimensional 

UGS  unmanned  ground  system 

UGV  unmanned  ground  vehicle 

UFV  unmanned  fire-fighting  vehicle 

yd  yards 
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